This disclosure relates to multi-point and multi-touch enabled devices in general, and more specifically to reducing the amount of latency between touch events in multi-point and multi-touch enabled devices and the corresponding updates to the display pipeline of such devices.
Multi-touch enabled devices are known in the art. A multi-touch enabled device is a device that can sense multiple touches at the same time. Thus, a multi-touch enabled device can, for example, sense two touch events that take place simultaneously at two different positions on a multi-touch panel and are caused by two fingers being pressed down on the panel. Examples of multi-touch enabled devices are discussed in U.S. Pat. No. 8,970,501, entitled “PROXIMITY AND MULTI-TOUCH SENSOR DETECTION AND DEMODULATION,” filed on Jan. 3, 2007, and hereby incorporated by reference in its entirety (hereinafter, “the '501 patent”). Multi-point enabled devices define a more general set of devices that include multi-touch enabled devices, as well as similar devices such as the multi-proximity sensor devices discussed in the '501 patent mentioned above.
While the benefits of multi-touch enabled interfaces are known, these devices can present some interface and graphics processing pipeline design challenges. More specifically, if a multi-touch interface is being used, two or more touch events can simultaneously or near-simultaneously occur at different portions of the display. This can make it difficult to split the display into different portions and have different independent software elements process interactions associated with each portion. Furthermore, even if the display is split up into different portions, multiple touch events can occur in a single portion. Therefore, a single application, process, or other software element may need to process multiple simultaneous or near-simultaneous touch events.
However, each application, process, or other software element needs to consider multiple touch interactions, so the relative overall cost and complexity of software running on the multi-touch enabled device may be quite high. More specifically, each application may need to process large amounts of near-simultaneous incoming touch data. This can require high complexity in applications responding to the incoming touch data and can result in undesirable latency between when a user touches a display and when the display is updated appropriately to reflect the user's touch act(s).
In one example, the display screen may be refreshed at a rate of 60 Hz, meaning that the “vertical blanking interval” (VBI) of the display screen is every 16.7 ms. (It is to be understood that, with a faster screen refresh rate, e.g., 120 Hz, the VBI would be decreased correspondingly, e.g., 8.333 ms, in the case of a 120 Hz refresh rate.) This means that, within some number of multiples of the VBI interval, the graphics processing pipeline needs to: process the incoming touch data; have the relevant application(s) process and handle the incoming touch data appropriately; animate the resulting changes to graphical elements (e.g., using an animation API for drawing 2D and 3D graphics, such as OPEN GL®); render the resulting content using a GPU or other graphics-specific processing unit; and write the rendered output to a frame buffer for display on the display screen at the beginning of the next-occurring VBI. (OPEN GL® is a registered trademark of SILICON GRAPHICS, INC.) One additional constraint includes the fact that most multi-touch enabled displays cannot run their multi-touch receive operation at the same time as their multi-touch scan operation (i.e., the device cannot simultaneously be receiving new touch input and attempting to determine where the immediately previous touch input was located).
As may now be more clearly appreciated, the longer that the processing; handling; animating; and rendering steps take to complete, the greater the number of VBIs that will pass between when the user touch event took place on the device display and when the display is actually updated to reflect said user touch event. This lag or “latency” between touch input and the corresponding display update response is undesirable in most situations. Therefore, what is needed is an intelligent manner of speeding up the multi-touch graphics processing pipeline that will reduce latency between touch input and display updates, but will not unfavorably affect the graphical UI experience for the user of the device.